Web Crossing/Pro Sysop Documentation

» Redundant database architecture

Redundant database with 2 mirrors

All of the servers, including the source, can be used to process user requests. In very high-volume sites, the source may be dedicated to synchronizing the various mirror databases.

Each mirror may run as a separate process on the same machine as the source, or on its own machine. Each server, source or mirror, keeps its own copy of the discussion database. TCP/IP connections from the source to each mirror are used to synchronize all of the databases.

The mapping of a user request to a particular Web Crossing server is done outside of Web Crossing. For example, you could use DNS rotation to have domain-name lookups rotate among the various servers, or hardware such as Cisco's LocalDirector to transparently assign each request to an available server.

Because each server keeps its own copy of the database, most requests can be served locally, without accessing the source database. Requests that change the database, such as checking subscriptions or changing user preferences, are passed to the source and then from the source to the other mirrors. Requests that create a new database object, such as posting a message or creating a new discussion, are passed to the source to have the object's unique ID assigned centrally, and are then passed by the source to all mirrors.

» Configuring redundant servers

Each redundant server must have its own license certificate.

» Performance scaling

Because each post has to be forwarded to the source server for processing, and then back out to all the mirrors to keep their databases synchronized, each server is processing all of the new messages submitted to the site.

With 20:1 views to posts and 10 servers, each server is spending about 1/2 of its time adding new messages to the database. (This is a conservative estimate, because updating the database is quite a bit cheaper than building the response to a user request.) So at 20:1 views to posts, 10 servers will provide 5 times the throughput of a single server. If one of your servers can do 300,000 hits per day, then 10 servers will scale to 1.5 million hits per day.

» Operations with redundant servers

Configure the additional mirror server, either as another process on an existing machine or on a new machine.

On the source server, use the Distributed servers panel to add the mirror server.

On the mirror server, use the Distributed servers panel to configure the mirror and connect it to the source.

Note that the source and mirror synchronize by copying the source's DB over the TCP/IP connection. Copying the source's database can take the source off-line for a few minutes, and it can take a few minutes for the mirror to receive the database and start processing requests. If possible, it is desirable to take the source and mirror off-line during this synchronization. For example, if you are using DNS rotation, the source and mirror Web servers could be configured to transparently forward requests to another server.

You can check on the progress of the synchronization by logging in to the mirror machine as the sysop. For the sysop, all pages include the current synchronization status at the top of the page (except pages served through the webx.tpl file).

The first step is to remove the mirror server from the request rotation. With DNS rotation, this means configuring the mirror's Web server to forward the mirror's requests to another machine.

Once the mirror is off-line, you can shut it down normally, as if it was a stand-alone Web Crossing server.

Moving the source server to one of the mirrors requires that service be shut down for a few minutes. After service is shut down and user requests are no longer being served, you can shut down the source server, configure one of the mirrors to be the source, and then configure the remaining mirrors to connect to the new source.

The new source will synchronize each of its mirrors to its database, and service can then be resumed.

With a large database, it can take a few minutes to make a backup copy. The recommended technique is to temporarily remove one of the mirror servers from the rotation for user requests, do the backup on that mirror server via the sysop Backup command, and then bring that mirror back into service.

The Backup command makes a backup of the current database called webxdb.1. This backup file can be copied to external storage to allow last-resort restarting of the discussion service.

If a mirror server shuts down unexpectedly, you can simply remove it from the rotation. If you are using DNS rotation, this requires a spare machine to forward requests to an available server; if you are using Cisco's LocalDirector or equivalent, the mirror server will be removed from service automatically.

If the source server shuts down unexpectedly, you need to make one of the available mirrors into the new source as described above. When the old source comes back on-line, you can reconfigure it to become a mirror of the new source.

» Shared user directory

You configure shared directory service through the Distributed servers sysop control panel.

» The Excite search engine

The Excite search engine builds an index of your discussion database, including all messages, discussion headings, and folder headings. Subsequent search requests use this index to dramatically speed up the search operation. The Excite index is a small percentage of the discussion database size, typically taking less than 10% additional disk space.

In addition to performing very high-performance searches, the Excite search engine automatically builds "concept based" document clusters. Documents with many similar words and phrases are clustered together, and may be returned by a search as related documents, even though they do not contain the exact keywords used in the search request.

The search engine index must be rebuilt periodically in order to incorporate new messages and discussions. Web Crossing can automatically schedule index builds daily or when some number of new messages have been posted. The index rebuild takes place in the background, so that your Web Crossing conference continues to run (and search with the old index) during this process.

New messages will not be found by the Excite search engine until the index is rebuilt. You can tell Web Crossing to use full-text keyword searching on these new messages until the index is rebuilt, so that the full database is always searched.

» Managing the Excite search engine

To rebuild an index manually or check on the current index rebuild status, see the commands at the end of the Sysop Controls panel.

When you install Web Crossing/Pro, a Search directory is created in the same directory as the Web Crossing program. This Search directory contains three files used to build the search index: stop.ptr, stop.key, and stem.tbl. The index is built into Search directory as a collection of files named Search.xxx. The next index is built as a collection of files named Backup.xxx. Once the next index is completely rebuilt, the old Search files are deleted and replaced by the new index.